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® An enhanced input-output element has a self- 
timed interface (STI) in which a clock signal clocks 
bit serial data onto a parallel, electrically conductive 
bus and the clock signal is transmitted on a separate 
line of the bus. The received data on each line of the 
bus is individually phase aligned with the clock sig- 
nal to provide a cost effective, modular, broadband, 
input/output element that can serve economically 
two channels and is modularly scalable to sen^e 
several hundred channels. 
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This inv ntion relates to an improved method 
and apparatus for transmitting digital data at high 
speeds via a parall I data bus. and mor particu- 
larly, to a method and apparatus that provides a 
cost effectiv modular, high banded input/output 5 
element that can serve economically a few chan- 
nels and is modularly scalable to serve several 
hundred channels. 

TTie present United States patent application is 
related to the following co-pending United States 10 
patent applications incorporated herein by refer- 
ence: 

Application Serial No. 262.081, filed 06/17/94 
(attorney Docket No. P09-93^). entitled "Digital 
Phase Locked Loop with Improved Edge Detector." 75 
and assigned to the assignee of this application. 

Application Serial No. 261.515, filed 06/17/94 
(attorney Docket No. PO9-93-054), entitled "Self- 
Timed Interface." and assigned to the assignee of 
this application. ^ 

Application Serial No. 261,522. filed 06/17/94 
(attomey Docket No. PO&-93-056), entitled "Mul- 
tiple Processor Unk." and assigned to the assignee 
of this application. 

Application Serial No. 261,603, filed 06/17/94 
(attomey Docket No. PO9-93-058). entitled "Mas- 
sively Parallel System." and assigned to the as- 
signee of this application. 

Application Serial No. 261.523. filed 06/17/94 
(attomey Docket No. PO9-93-059). entitled "At- 30 
tached Storage Media Link," and assigned to the 
assignee of this application. 

Application Serial No. 261,641, filed 06/17/94 
(attomey Docket No. PO9-93-060), entitled "Shared 
Channel Subsystem." and assigned to the assign- 35 
ee of this application. 

As will be appreciated by those skilled in the 
art. such factors as noise and loading limit the 
useful length of parallel busses operating at high 
data rates. In the prior art, the length of the bus 4o 
must be taken into account in the system design 
and the bus length must be precisely as specified. 
Manufacturing tolerances associated with physical 
communication link (chips, cables, cord wiring, 
connectors, etc.) and temperature and variations in 45 
power supply voltage also limit the data rates on 
prior art busses comprised of parallel conductors. 
Further, many prior art computer systems transfer 
data synchronously with respect to a processor 
clock, so that a change in processor clock rate may 50 
require a redesign of the data transfer bus. 

General purpose computer systems are com- 
prised of central processing element(s), local mem- 
ory (storage) elements and input/output (I/O) le- 
ments. Of . these, the processing and storage ele- 55 
m nts hav b n fitted directly from the continuous 
advances being mad in circuit d nsity. enabling 
more and more processing capability/storage per 



chip. 



Th I/O element, gated by other physical con- 
straints, has not been abl to track the performanc 
gains of the processing/storag lements. Hence, 
the I/O element can limit the overall computer 
systems' performance in many applications. Fur- 
ther aggravating this problem are new system ap- 
plications, such as client/server, that require large 
numbers of I/O. and multimedia that demand high- 
er perfomiance I/O elements. It also follows that for 
many applications, the smaller and smaller, yet 
faster, central processor elements (benefitting from 
circuit technology density advances) require more 
and more I/O capability to satisfy the increased 
demand for data (to be moved and/or to be pro- 
cessed). These trends can be expected to persist 
Given that a primary role of a computer system I/O 
element is that of a transformer of bus speed (as 
well as of protocol and technology) between in- 
creasingly faster processor/storage elements and 
the slower, relatively unchanging speed of the at- 
tached I/O controllers, plus increases in required 
connectivity will continue to drive the need for 
improved efficiency and perfomnance of the I/O 
element. 

The I/O element often takes the fom>. espe- 
cially on the most powerful systems, of a large 
multiplexer feeding and being fed by a network of 
more numerous and slower buses/links (becoming 
more numerous and slower the further away from 
the host they are in the network). 

A current typical large computer system I/O 
element can require more than 100 channel func- 
tions as the middle stage/level in a hierarchically 
arranged busing networi< (between the highest level 
internal bus (fastest) and the more numerous and 
slower I/O controllers). The channel functions are 
packaged as part of the computer system and 
provide the interfaces to communicate, over dis- 
tance (km), with the I/O controllers. 

The large number of channel functions re- 
quired, the channel transceiver design, the power 
required for distance and the connectors needed 
for extemal cable attachment all impede I/O ele- 
ment miniaturization. 

In the prior art system, efforts are made in the 
I/O element design to avoid requiring excessive 
host signal pins (obtainable only through exotic/ 
expensive packaging). Toward that end, minimum 
width data path/bus connections are employed be- 
tween the channels and higher levels (internal bus- 
es) of the networi<. Even so. the number of signal 
pins, intemal cables, connectors and circuit board 
compi xity remains high. 

The narrow buses are cycled faster than wider 
buses to achieve equival nt data transmission per- 
formance: how V r. this r stricts maximum bus 
(cable) length. This trade-off betwe n cabi dis- 
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tance/net length and cycle tim is typical with most 
pnor art bus/link technologi s and is a significant 
packaging constraint. Optimal packaging for cost 
and extendibility Is compromised by requiring all 
channels to be within a limited (e.g.. 3 meters) s 
cable distance from the processor storage/control 
elements. This is especially troublesome in the 
more powerful systems where the system package 
must accommodate 100 or more channels. 

Then too. prior art systems operate the nu- w 
merous channelsAjuses synchronously to the host 
system clock to avoid inflating host circuit count via 
additional synchronizing buffers and logic. The 
channels/buses operating cycle time ranges are 
slower than the host, thus clock multiplier logic is is 
used to provide clocks that are a multiple of the 
host clock cycle. This adds complexity to the de- 
sign (as different multipliers can be required when 
host cycle times change) to remain within chan- 
nel/bus operating ranges. 20 

An object of this invention is the provision of a 
cost effective bus data transfer system that can 
operate at high data transfer rates without tight 
control of the bus length, and without system clock 
constraints; a system in which the maximum bus 25 
length is limited only by the attenuation loss in the 
bus. 

Another object of the invention is the provision 
of a general purpose, low cost, high performance, 
point to point data communication link where the 30 
width and speed of the interface can easily be 
modified to tailor it to specific bandwidth require- 
ments and to specific implementation technologies 
including VLSI technologies. 

A further object of the invention is the provision 35 
of a bus data transfer system that operates a clock 
speed equal to the data rate. 

A more specific object of the invention is the 
provision of a system that adjusts the phase or 
amval time of the incoming data on the receive 4o 
side so it can be optimally sampled by the local 
receive clock, compensating for many of the manu- 
facturing tolerances associated with the physical 
link (chip, cable, card wiring, connectors, etc.) as 
well as temperature changes and power supply 45 
output variations. 

Another object of the invention is the provision 
of an enhanced I/O element to facilitate decen- 
tralization of the I/O element into (n) sub^lements 
with significant advantage to system cost, flexibility 50 
and extendibility. 

One more object is to allow a customized array 
of different- channel adaptors to be implemented 
within the I/O element without impacting xpensiv 
host specific hardware and software. 55 

A still further object of the inv ntion is to en- 
abl each I/O sub-element to operate asynchro- 
nously to the host clock, thereby avoiding the clock 



control complexity associated with th previously 
described variable clock multiplier logic. This per- 
mits th I/O sub-el ments to be used by a wider 
rang of hosts/hostcycle times. 

Briefly, this invention contemplates the provi- 
sion of a self-timed interface (STI) in which a clock 
signal clocks bit serial data onto a parallel, elec- 
trically conductive bus and the clock signal is 
transmitted on a separate line of the bus. The 
received data on each line of the bus is individually 
phase aligned with the clock signal. The received 
clock signal is used to define boundary edges of a 
data bit cell. It is defined individually for each line 
and the data on each line is individually phase 
adjusted so that, for example, a data transition 
position is in the center of the defined cell. The 
data is read into a buffer where it is decoded and 
read out synchronously with the receiver system 
clock. At the data rates contemplated in the ap- 
plication of this invention, the propagation delay is 
significant. However, within limits, the buslength is 
not critical and is independent of the transmit and 
received system clock. The phase adjustment can 
compensate for a skew of up to one bit cell across 
the width of the bus. 

In one specified embodiment of the invention, 
data to be transmitted is transferred to a buffer 
synchronously with the transmitter system dock, 
which may or may not be the receiver system* 
clock. A controller formats the data into packets for 
byte parallel, bit serial, transmission along with 
headers specifically coded to provide unique data 
patterns that allow for conrection of skew of up to 
three bit cells in addition to the inrtiaJ phase adjust- 
ment. 

The foregoing and other objects, aspects and 
advantages will be better understood from the fol- 
lowing detailed description of a preferred embodi- 
ment of the invention with reference to the draw- 
ings, in which: 

Figure 1 is an overview block diagram illustrat- 
ing the application of a self-timed interface, in 
accordance with the teachings of. this invention, 
to data communication among computer chips. 
Figure 2 is a block diagram illustrating one em- 
bodiment of a transmitter serializer for imple- 
menting a self-timed interface in accordance 
with this invention. 

Figure 3 is a block diagram illustrating byte 
synchronization in accordance with ttie inven- 
tion. 

Figure 4 illustrates phase alignment and sam- 
pling logic in accordance witt) a preferred em- 
bodim nt of th invention. 
Rgure 5 illustrates an input/output el ment con- 
figuration in accordance with the t achings of 
this invention. 
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Referring now to Rgure 1 of the drawings, it 
illustrates one enDbodiment in which a self-timed 
int rfac in accordance with th teachings of this 
invention can be used. This ex mplary embodi- 
ment of the self-timed interface provides data com- 
munications between two microprocessor chips, 
labeled here as Chip A and Chip B. However, as 
will be apparent to those skilled in the art. the self- 
timed interface of this invention is applicable to 
provide data transfer between a wide variety of 
components or nodes. 

Chip A has a transmit port labeled 12A and 
Chip B has a transmit port labeled 12B. Similarly, 
Chips A and B have receive ports labeled 14A and 
148. respectively. The ports are connected by two 
self-timed interface busses 16; one for each trans- 
mission direction. In this exemplary embodiment of 
the invention, each bus 16 is one byte wide, and 
comprised of nine electrical conductors; eight con- 
ductors for data and one conductor for a clock 
signal. 

Each transmit port (12A and 12B) includes a 
transmit logical macro 18 that provides a logical 
interface between the host logic and the self-timed 
interface link 16. Sync buffers 22 provide an inter- 
face between the host clock and the self-timed 
interface clock. This allows the self-timed interface 
link to run at a predetermined cycle time that is 
independent of the host clock, making the self- 
timed interface link independent of the host. An 
outbound physical macro 24 serializes a word-wide 
data flow into a byte-wide data flow that is transmit- 
ted along with the clock on the self-timed irterface 
link 16. 

Each receive port (i.e.. 14A and 14B) includes 
an inbound physical macro 26 that first dynamically 
aligns each data bit with the self-timed interface 
clock signal. It aligns any bits with skew up to three 
bit cells and deserializes the bytes into words. A 
receive togical macro 28 provides an interface be- 
tween the self-timed interface receiver logic and 
the host logic and generates link acknowledge sig- 
nals and link reject signals, which are coupled by 
internal links 33 and transmitted back to tfie trans- 
mitting port via an outbound self-timed interface 
link 16. In order to compensate for variations in 
lectrical path delay, the phase of the incoming 
data is adjusted, or self-timed. Each bit (line) is 
individually phase aligned to the transmitted refer- 
ence clock and further aligned to compensate, 
wittiin embodiment, for up to three bit cells of skew 
between any two data lines. The self-timing opera- 
tion has ttiree parts. The first is to acquire bit 
synchronization; tiie second is byt /word align- 
ment; and the third is maintaining synchronization. 

In acquiring bit synchronization, th link tak s 
its If from a compi t ly untimed stat into synchro- 
nous operation. Any pr vious condition on the STI 



interface or logic is disregarded with a complete 
logic reset. The bit synchronization process can be 
rapidly established, for exampi on the order of 
200 microseconds. The phase of tiie incoming data 
5 IS manipulated on a per line basis until the data 
valid window or bit interval is located. This is 
accomplished using a phase detector that locates 
an average edge position on tiie incoming data 
relative to the local clock. Using two phase detec- 
10 tors one can locate two consecutive edges on data 
and tiiese two consecutive edges define tiie bit 
interval or data valid window. The data to be sam- 
pled by the local clock is the phase of the data 
located halfway between the two edges of the data. 
75 Byte alignment takes place by manipulating the 

serial data stream in whole bit times to properly 
adjust the byte position relative to a deserializer 
output. Word alignment takes place next by man- 
ipulating the deserializer data four bit intervals a 
20 time to ensure proper word synchronization on tiie 
STI interface. A timing sequence allows proper bit. 
byte and word synchronization. 

Synchronization maintenance occurs as part of 
tiie link operation in response to temperature and 
25 power supply variations. 

Referring now to Figure 2. which illusti-ates one 
embodiment of a fransmit serializer for a bit serial 
byte parallel interface used in tiie practice of the 
invention. Here a four byte wide data register 23 
30 receives parallel inputs 25 (bytes 0. 1, 2 and 3 
inputs shown here) and multiplexes tiiem to a one 
byte wide self-timed interface. Data is clocked from 
the register 23 by a divide by two of tiie self-timed 
interface clock on line 27. Bit zero from bytes 0. 1. 
35 2 and 3 are serialized and ti^ansmitted on link 0 of 
the self-timed interface, shown here. Bit 1 from 
bytes 0. 1. 2 and 3 will be transmitted on link 1 (not 
shown)and so on. 

To minimize tiie bandwidtii requirements of tiie 
40 communication media the STI clock is one half the 
frequency of tiie transmitted data (baud) rate, i.e.. a 
75 Mhz clock will be used for a 150 Mbit/S data 
rate. The clock will be generated from an STI 
osdilator source, tfiis is done to decouple tiie sys- 
45 tem or host clock from tiie STI link. The data will 
be ti-ansmittedwith botti edges of the clock. 

Refen-ing now to Figure 3. assuming a bit syn- 
chronization process as described in connection 
with co-pending United States patent application, 
50 Application Serial No. 261 ,515, filed 06/17/94 (attor- 
ney Docket No. PO9-93-054). entitled "Self-Timed 
Interface." and assigned to the assignee of tiiis 
application, byte synchronization starts by coupling 
th phase aligned data (now 2 bits wide) into a shift 
55 register 33 whose outputs are coupled to mul- 
tiplexer 35. The conti-ol inputs to th multiplex r 
will b used to desk w tiie particular data line from 
the other data lines by whole bit times. Th de- 
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serializer data output for a particular data line is 
monitored for tfie expected timing pattern pc 0 1 0 
wher X is a don't care) to determin tti proper 
order of the rec ived data. If at any time a zero is 
detected in th bit 3 position the multiplexer is 
incremented thus moving the byte boundary by 
one bit time. This process is repeated until the 
proper byte boundary is located. The multiplexer 
control wraps around from a binary 3 to a binary 0 
in case the correct position was incorrectly passed 
through the previous time, this function allows syn- 
chronization of data lines skewed by more than an 
entire bit time. 

Referring now to Figure 4, in this embodiment 
of the invention, the clock rate is the same as the 
data rate. The data edges that define a data win- 
dow are each detected independently of the other 
and the data is sampled at the midpoint between 
the edges when the edges have been aligned with 
the clock. The position of the edges of increment- 
ally separated phases of the input data stream are 
successively compared to the position of the rising 
and falling edges of the clock in order to locate the 
edges of the data stream with respect to both 
edges of the clock (e.g.. the rising and falling 
edges). 

The data phase pairs are generated in this 
specific embodiment of the invention by three in- 
crementally selectable delay elements 80. 82. and 
84. For example, the elements 80 and 82 provide 
delays, respectively, in 1/10th and 1/5th bit time 
increments and element 84 provides fine incre- 
ments on the order of 1/20th of a bit time. The fine 
delay element 84 is separated into three groups to 
provide early edge detection, system data detec- 
tion, and late edge detection. An early guard band 
selector 86 successively selects one phase of the 
data stream to provide an "early" phase of the 
incrementally separated phases - one for the rising 
edge -and one for the falling edge. Similarly, a late 
guard band selector 90 successively selects one 
phase of the data stream to provide a "late" phase 
of the incremental phases - again one for the rising 
edge and one for the falling edge. A selector 88 
selects incremental phases for the mid-cell system 
data position. 

A selected data phase is coupled as an input 
to master-slave RES-FES latch pairs 92. 94. and 
96. The rising edge data samples are clocked into 
the RES latches and the falling edge data samples 
are clocked into the FES latches. The outputs of 
the RES-FES latch pair 92 are connected to an 
early edge, detector 98. Similarly, the outputs of the 
RES-FES latch pair 96 ar coupled to a lat dg 
detector 100. The RES latch of pair 94 is coupled 
to the early edge detector 98 and th FES latch of 
pair 94 is coupled to th lat edge detector 100. 
Each edge detector (98 and 100) outputs a "lead". 



a "lag" or a "do nothing" output which indicates 
the location of a data edge with respect to the 
ref r nc clock edg location. Th output of each 
dge detector is coupled via a suitable filter 102 
5 (i,e.. a random walk filter), back to its respective 
selector 86 and selector 90, respectively. The se- 
lectors shift the phase of the data coupled to the 
RES-FES latches in the direction indicated, or if 
"do nothing" is indicated, the phase of the data at 
10 that edge is not shifted. 

Data control logic 104 controls the system data 
output by selecting tiie phase of the data that is 
halfway between tf^e two data edges when the data 
edges are aligned with the reference clock. A 
75 phase of the data (Data 1 and Data 2) is outputted 
at each reference clock edge. 

In operation of a specific embodiment, at pow- 
er on the logic will automatically begin the bit 
synchronization process. A 16 microsecond (final 
20 value TBD) timer is started, the bulk delays are 
reset to their minimum delay and a 16 bit counter 
running off the divided down clock is started. The 
edge detect circuitry will sample the incoming data 
with tfie received reference clock. TTie edge detec- 
25 tor will output a "lead", a "lag" or a "do nothing- 
signal that indicates the data edge location relative 
to the reference clock. This signal is filtered by a 
Random Walk Filter (RWF) and fed-back to the 
selectors of their respective RES and FES circuits. 
30 The selectors shift the phase of the data into the 
RES and FES as indicated by tiie edge detector. 
Each edge detector operates independentiy of the 
other. Each will locate the transitions on data rela- 
tive to the received (ref) clock by manipulating the 
35 incoming phase of the data into the edge detector 
as described above. The phase of the system data 
is controlled by the data control togic which selects 
the phase of the data halfway between the two 
edge detectors. In parallel with tiie'brt synchroniza- 
40 tion process, the order of bits out of the de- 
serializer are manipulated to the correct order (see 
byte^vord synchronization below). When the 16 
microsecond timer trips the algorittim resets a de- 
serializer error latch and restarts the 16 micro- 
45 second counter. The deserializer output is com- 
pared against the expected timing pattern pc 0 1 0 
where X is a don't care). A single miscompare on 
any cycle during the next 16 microseconds will set 
the deserializer en-or latch. When the 16 micro- 
50 second counter trips again the algoritiim checks 
tile addresses of tiie EGB. LGB. and data selec- 
tors, deserializer error latch. In order for a bit to 
end the initial bit synchronization search state, the 
d serializ r output latch must have remain d r set 
55 AND tiie all selectors must be properiy center d in 
ttieir tracking range (centering nsures that adjust- 
m nts can b made to allow for the fracking of 
temp, and power supply variations after tfie initial 
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bit synchronization process). If both conditions are 
not met then the algorithm adds a bulk d lay 
element, resets th 16 microsecond counter and 
the search process begins once again. Each and 
every bit (data line) on the STI interface undergoes s 
this process in parallel. Once an individual data line 
is determined to meet the initial bit synchronization 
criteria described above it is degated white the 
other lines continue to be adjusted. The bit syn- 
chronization process is complete once all bits are ro 
adjusted and meet the search criteria The logic 
will not exit the bit synchronization mode until the 
16 bit counter trips. 

Rnally word alignment takes place. Word align- 
ment is established by manipulating the de- 75 
serializer output bus four bits at a time until word 
synchronization is established. Note that the first 
register is shifted by four bit times relative to the 
second register. Four bit times Is the maximum any 
data bit can be skewed relative to another data bit 20 
(3 bit times on link + 1 bit time from phase 
alignment section). 

During normal operation the physical macro 
will continuously monitor the incoming data to en- 
sure that the optimum clock sampling relationship 25 
exists. Small updates will be made to track tem- 
perature, power supply and data jitter. These up- 
dates will be seamless and transparent to the host 
logic. Approximately 1/2 a bit time of delay will be 
needed to compensate for temperature and power 30 
supply variations to maintain proper synchroniza- 
tion. This added delay is in the fine delay elements 
section. There is also circuitry to monitor tHe posi- 
tion of the guard bands relative to the allowable 
range of operation. If a guard band rpaches the end 35 
of its range, two cases exists: 1) a new bulk delay 
element is added and the fine delay elements are 
adjusted accordingly. Note this can cause sampling 
errors in the data. The circuitry that makes these 
on the fly bulk adjustments can be inhibited so no 4o 
on the fly bulk delay adjustments are made during 
normal operation. The second case exists when 
one of the guard bands reaches the end of its 
range and the on the fly bulk delay adjustment is 
inhibited, the physical macro will signal the logical 45 
STI macro that a bit synchronization is required 
soon. The link should finish the immediate work 
and force the link into timing mode. 

Referring now to Rgure 5. a host processor 60 
in this exemplary embodiment needs only eight 50 
STI ports 62 to support up to 256 input-output 
channels 65, which are in turn coupled to input- 
output controllers 65. STI links 66 couple the ports 
62 to input-output sub-elements 68 labeled lOSE. 
Th sub-elements 68, which may be integral with 55 
or physically located away from the host 60, ach 
can support up to 32 channel functions. The ex- 
tended operating length of the self-timed interface 



66 (i.e., up to ten met rs without perfomnance 
degradation) makes it possible to package this 
many chann I functions in a cost effectiv manner. 

A single STI Is capable of supporting 32 chan- 
nel functions. For those system configurations re- 
quiring only up to 32 channels therefore, only one 
STI port, cable and I/O sub-element need be used. 
This enables incremental cost advantages over cur- 
rent centralized I/O elements in which even the 
smallest channel configuration can incur the cost of 
hardware in place for the maximum (e.g., 256) 
channel configuration. 

STI enables each I/O sub-element to operate 
asynchronously to the host clock, thereby avoiding 
the clock control complexity associated with the 
previously described variable clock multiplier logic. 
This permits the I/O sut>-elements to be used by a 
wider range of hosts/host cycle times. Because of 
the high performance of STI. fewer are needed, 
thus the re-synchronizing buffers/logic at each end 
is not burdensome. 

While the invention has been described in 
terms of a single prefered embodiment, those 
skilled in the art will recognize that the invention 
can be practiced with modification within the spirit 
and scope of the appended claims. 

Claims 

1. An input-output element for coupling digital 
data at high data rates between a host proces- 
sor and a plurality of input-output controllers, 
comprising in combination: 

a host processor; 

a high-speed, self-timed interface; 

an input-output channel sub-element; 

a plurality of input-output channels con- 
necting said input-output sut)-element respec- 
tively to a plurality of input-output controllers; 
said high-speed, self-timed interface connect- 
ing said host processor to said input-output 
channel suthelement; 

said high speed, self-timed interface including 
a transmitting node for transmitting a digital 
data and a clock signal and a receiving node 
for receiving said digital data and said clock 
signal, said transmitting node connected to 
said receiving node by a parallel data bus to 
individual lines of which a digital data stream is 
coupled by said clock signal at said transmit- 
ting node, 

said bus including a separate line for transmit- 
ting said clock signal, and said receiving node 
including means to phase align a digital data 
stream on each of said lines separately with 
respect to said clock signal. 
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2. An input-output elennent as in claim 1 wh rein 
said input-output sub- lement and said self- 
timed int rface ach support up to 32 chan- 
nels, 

5 

a An input-output element as in claim 1 wherein 
each input<»utput sub-element operates asyn- 
chronously with respect to a host clock signal. 
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